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ABSTRACT 

V This dociMtient is a monograph in a personnel 

development series addressing issues that are pertinent for 
policy-making personnel concerned .with program evaluation and 
\ personnel preparation. The major intent of the monograph is to 
present a variety of evaluation practices and procedures used in the 
Wolution of a vocational/special education personnel preparation 
program. As a means of illustrating and discussing the major 
evaluation points in a program, a three-phase view of program 
evaluation is delineated. Section 1 considers planning evaluation 
(de^termining program goals and selecting the program strategy). 
Section 2 describes process evaluation focusing on the implementation 
aspects of a program. Three case examples describe evaluation 
activities conducted for a short-term workshop, a course, and an 
inservice project. Section 3 examines impact evaluation that looks at 
the outcomes and effects of a program on its graduates and their 
respective work environments. Two case examples of impact evaluation 
designs in vocational/special education personnel preparation 
projects are highlighted. A final section suggests procedures and 
guidelines to increase use of evaluation findings. (YLB) 
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FOREWORD 

u Over the past decade the problems and difficulties that face hanpiicapped 
youth in their efforts to obtain and maintain employment have been widely 
documented by researchers, public policy analysts, and advocacy organiza- 
tions. In the 1970s the U.S. Congress enacted several pieces of education, 
training, and employment legislation to focus, in part, on resolving these 
problems. The Education for All Handicapped Children Act of 1975, along 
with the Vocational Education Amendments of 1976, the Comprehensive 
Employment and Training Act of 1978, and several civil rights initiatives, 
placed priority upon assuring that handicapped youth receive appropriate 
vocational education programs and services. These various pieces of legisla- 
tion acknowledged the concurrent need for staff development and teacher 
education programs to assure that effective programs and services are de- 
livered. Within the vocational education, special education, rehabilitation, 
and CETA systems there are nearly a million prof^ssionals--the vast majority 
of whom have limited or no expertise in planning and providing comprehensive 
vocational programs and services for disabled youth and adults. The need 
for training programs to update teachers, support personnel, counselors, 
coordinators, and administrators is great. There is also an enormous need 
for training other individuals (such as employers, parents, advocates, co- 
workers, non-disabled peers) if youths with special needs are to be success- 
ful in their transition from school to work. 

Planning and conducting effective personnel development programs that 
serve the career development needs of handicapped youth involves a variety 
of complex tasks. Developing appropriate interagency, collaborative training 
arrangements is essential to insure that currc.it knowledge and expertise is 
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Utilized from the fields of vocational education, special education, rehablita- 
tion, career development, and employment and training. Decisions must be 
made relative to the specific training needs of the target audience. Fre- 
quently, the needs of inservice practitioners must be considered along with 
the needs of trainees who are preparing to enter the field for the first time. 
The question of student needs is also present. The process of providing 
vocational education for severely handicapped youths is, by nature cf the 
students served and the training technology, considerably different from 
training mildly handicapped youth. Other critical dimensions related to the 
content of personnel development encompass such areas as: vocational assess- 
ment, career guidance, and evaluation of training programs. The need for 
and patterns of personnel certification in the field of vocational/special educa- 
tion is also a continuing concern for personnel development programs. 

During 1980-82 the University of Illinois hosted a series of three confer- 
ences which focused upon improving personnel preparation programs in voca- 
tional/special education. These conferences were conducted as part of the 
Leadership Training Institute/Vocational and Special Education, which was 
supported by a grant from the Division of Personnel Preparation, Special 
Education Programs, U.S. Department of Education. As individuals responsi- 
ble for personnel preparation programs in vocational/ special education met 
and shared their experiences and concerns, a clear need emerged for a series 
of monographs on designing, implementing, and evaluating personnel develop- 
ment programs. The need to address the crjitical questions and identify 
effective policies and practices related to personnel development was obvious 
following the initial conference held in Champaign, Illinois in April 1980. The 
project staff used a small advisory group of individuals attending the confer- 
ences to outline the Perspectives monograph series. Needs assessment data 

ii 
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collected duriiig and prior to the first conference was used by the group in 
identifying the major topics to be addressed in the series. S^aff involved in 
the vocational/career education projects funded by the Division of Personnel 
Preparation were then invited to become members of the various monograph 
writing teams. Under the expert .guidance of Dr. Janet Treichel, LTI Trainr 
ing and Dissemination Coordinator, the writing teams formulated their mono- 
graphs to focus on such core components as: present state-of-the-art, 
effective policies and practices, and guidelines for personnel development 
programs. Dr. Treichel coordinated the planning and preparation of the 
series in a highly exemplary manner. Her leadership, commitment to excel- 
lence, and professional insight were valuable assets in editing this series. 

The monograph topics in the Perspectives on Personnel Development 
series include: Special Populations/Severely and Moderately Handicapped, 
Certification, Program Evaluation, Effective Interagency/1 nterdepartmental 
Coordination, Inservice Personnel Development,, Vocational Assessment, Pre- 
ser/ice Personnel Preparation, and Career Development/Guidance. 

We anticipate that the monographs will be useful resource documents for 
a variety of audiences. Teacher educators and administrators in higher 
education will find the series helpful in planning both preservice and inser- 
vice programs for special educators, vocational educators, counselors, educa- 
tional administrators, rehabilitation specialists, and others. State education 
agencies involved in certification, personnel development, and program admin; 
istration will find strategies, and suggestions for reviewing, evaluating, and 
formulating teacher training efforts in local agencies and universities. The 
monographs are also a rich source of ideas for parent and advocacy groups 
and professional associations as they seek to improve the knowledge and 
competence of personnel serving handicapped youth. o 

Hi 



This series represents a significant compilation of important and timely 
perspectives on personnel development in vocational/special education. It 
contains the wisdom and insight of nearly 50 leaders in the field. We feel it 
will be a valuable and important r esource in impro\4ng the- "appropriateness" 
of the programs and services received by the handicapped youths of our 
nation. 



L. Allen Phelps 
Director 

Leadership Training Institute/ 
Vocational and Special Education 



George Hagerty 
Project Officer 

Division of Personnel Preparation 
U.S. Department of Education 



6 



IV 



ERIC 



PREFACE 

The Perspectives on Personnel Development series has become a reality 
(4ue to the efforts of a number of individuals. These people were highly 
instrumental in the development, planning, and publication phases of the 
monographs. 

Appreciation and gratitude is extended posthumously to Margaret (Meg) 
Hensel. Meg was actively involved in assisting in planning for the personnel 
preparation conferences and the initial developmental stages for this series. 
We will continue to miss her enthusiasm and dedicated efforts. 

The LTI is indebted to Drs. Leonard Albright, University of, Vermont, 
and Geraldine Markel, University of Michigan, for their excellent work in 
developing this rrjonograph. This document addresses a number of issues that 
are pertinent for policy-making personnel concerned with program evaluation 
and personnel preparation. 

The reviewers for the Perspectives series also made important and signi- 
ficant contributions. Dr. Gary Clark of the University of Kansas reviewed 
each monograph in the series. Dr. Robert Brinkerhoff of Western Michigan 
University and Dr. Dave Pucel of the University of Minnesota served as 
reviewers for the Perspectives on Program Evaluation monograph. Their 
insightful comments and suggestions were very helpful in the preparation of 
the monograph. 

Sincere appreciation is expressed to Ms. Alicia Bollman, Ms.* Nancy 
Verbout, and Ms. June Chambliss for their dedicated efforts and patience in 
providing the secretarial expertise, necessary to produce this volume. 

Janet Treichel, Editor^ 
Coordinator, Training 'and Dissemination 
Leadership Training Institute/ 
Vocational and Special Education 
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The role of prograjp evaluation in the vocational/ special education person- 
nel preparation process has increased significa^ntly , due mainly to two influen- 
tial developments affecting the field. First, the relatively recent expansion of 
training programs in vocational/special education has generated greater profes- 
sional interest in tKe processes and outcomes of program evaluation in this 
area. Such information should, for example, be useful to program developers 
in determining the efficiency ^of their efforts and also keep them current with 
effective programs and practices across th(* country. Secondly, the tighten- 
ing of fiscal resources at federal and state levels will likely result in keener 
competition for shrinking program support funds. Consequently, the decision 
makers in these funding agencies may place an even gre?ter emphasis on 
program effectiveness- information. The collection and reporting of such 
information to these decision makers could hold special significance during 
this '^do jnore with less" fiscal era. 

The majoV intent of this monograph is to present a variety of evaluation 
practices and procedures used in the evolution of a vocational/special educa- 
tion personnel preparation program. As a means of illustrating ?nd discussing 
the major evaluation points in a program, a three-phase view of program 
evaluation is delineated. The three phases are: 

1. Planning evaluation , which is concerned with program design con- 
siderations; 

2. Process evaluation , which focuses on implementation aspects of a 
.program; and 

3. Impact evaluation , which looks at the influence of the program on 
its graduates and their respective v,. . environments* 

1 



This three-phase conceptualizatioiy of program evaluation is hardly a novel 
creation* Similar conceptual presentations have appeared in the literature, 
particularly within the past rive y^ars (e.g., StufflelDeam et a!., 1971; Phelps 
•6 Wentling, 1977; Morris & Fitz-Gibbon, 1 978;l Brirtkerhoff, 1980; Russo, 
1 980). . ' . 

The following descriptions of each of the three program evalu^^tion 
phases will provide the readei; with information relative to these questions: 
1 . What are the purposes of this type of evaluation? 
^2. At what point does this type of evaluation typically occur, in a 

program?. * , 

3. What are some example evaluation strategies used within this phase? 
^ 1. What sources could I ext)lo/e to obtain additional information on the 
particular idea expressed or example given? 

* 

While each phase .is treated in a separate manner, the underlying premise of 

If 

this paper is that evaluation is pervasive and ongoing, permeating all com- 
ponents of the personnel preparation program (Sanders S Cunningham, 1974; 
Skrtic, Knowlton, & Clark, 1979). 

Program evaluation is also a responsive activity, seeking to discern and 
accommodate the information needs of its audiences. By maintaining a respon- 
sive posture, the evaluation is of greater use to and more likely to be used 
by members of its audiences. This theme receives additional attention in the 
latter portion of this document. 



Phase I: Planning Evaluation 



Planning evaluation involves the identification and selection of program 

goals and strategies. More specifically, the primary purposes of the planning 

A* 

evaluation phase are: (a) to determine the proper goals of a personnel 

✓ »^ * 
preparation program, and (b) to help in selecting the most •appropriate 

strategy for achieving these goals ( Brinkerhoff, 1980)* 

Determining Program Goals , 

The first purpose of the planning evaluation phase, determining the 
proper program goals, is accomplished by analyzing the needs of the pro- 
spective training audience. For many preservice and inservice personnel 
preparation programs, especially beginning ones, the analysis has been done 
via the competency identification route (e.g,, Brolin o Thomas, 1972; Krantz 
S Weatherman, 1 976). In a review of 12 vocational/special education com-- 
petency studies, Albright (1978) noted that nearly all utilized a role analysis 
procedure for determining teacher competencies. This procedure essentially 
consisted of the following steps: 

~ Project ' staff and expert jury members review/select/edit 
competency statements ^ from existing lists; additional 
competency statements may be suggested by the re- 
viewers; ' 

> 

~ A list of competencies is synthesized and categorized 
through a consensus process; 

— Project staff designs a questionnaire, which includes a 
, listing of competencies and a rating scale for judging the 

Velativ? ^irr^portance . or criticality of each competency 
statement (other variables, such as frequency of use and 
need for additional training are sdmetimes included); and 

^ ' • •• . 

— The questionnaire is distributed to practitioners, (pp. 
:>1-22) 



A more recent example of this competency identification procedure appears in 
a final project report by VanNest and Barnhart (1980) who used the pro- 
cedure m dev^lOj.>ing a preservice curriculum for industrial and special educa- 
tion personnel at Kean College in^New Jersey, 

A second technique for analyzing needs, one that has recently gained 
prominence in light of intensified efforts in providing inservice in the voca- 
tional/special education area, is the needs assessment approach. While quite 
similar to the competency identification approach, the needs assessment pro- 
cess tends to be more focused in terms of its target audience, purpose, and 
informatiw scope. The needs assessment emphasizes identification and selec-- 

tion of inservice teacher training needs on the basis of highest priority 

* ♦ • 
order. A more global description of the needs .assessment process has been 

provided by Kaufman and Englisji (1979): 

the formal harvesting, collection and listing of needs, 
placing the neetis in priority order and selecting the 
needs -rrf highest priority for action,. The process in- 
clude the partners in planning, which in education are 
the learners, the implementers and the society • It re- 

^ quires ^that tl^ere js a consensus of the partners in the 
prioritization of n^eds and it strongly urges that the 
process include any additional external referent of sur- 
vival and contribution when determining need priorities, 

. Xpp/343-34il) 

The following is a sampljng of vocational/special education inservice 
needs assessment studies conducted at a variety of levels. 

Statewide: In terms of assessing teacher program needs on a stal^ewide 
{jasis, the studies conducted by Hughes (1978), Greenwood 
, ' and Morley v(1977), and Yung et al • (r'978) are often cited in 
the literature. The data collection procedure used in these 
studies was a mail survey instrument sent to vocational 
' * education personnel in the respective states (North Carolina, 

Iowa, and Arkansas),. 



University : Using a self-rated needs a cessment instrument developed by 
Phelps (1976) as one data collection source, Wentling and 
associates |i978) reported on the formative evaluation tech- 
niques employed in the development of a vocational special^ 
needs course. In their final project report, PeaJ< and Brown 
(1980) presented the conceptualization, design, ind pilot 



testing of a needs assessment survey irt^tjnjment in selected 

* \ . . 

secondary and post-secondary vocational schools in Minne- 

\ 

S9ta. 

Local: Regan and Deshler (1980) d^cribed the use of local steer- 

\ 

ing committees in conducting building-level needs assessment 
surveys as a major component of their inservice project. 
Mori (n.d.) utilized a survey instrument for collecting needs 
data from local district personnel. 



^Most exainple needs assessment activities mentioned' in the preceding 
paragraphs have used survey instruments to cglLect . needs data-. As sug-' 
gested by Kuh (1980), personnel doing needs assessments should consider 
including multiple data collection techniques. He stated: 

Many needs asisessment teams continue to rely on the district 
or building survey 'to document needs. There is no question that 
surveys can be effective and particularly efficient forms of gather- 
ing information from a large number of people. But a survey alone 
cannot document need . In practice^ most -needs assessments will 
' probably include some form of survey as one component of the data 
collection process. Most agree that information such as intervie'ws, 
* open forumsv and documentation or use of existing records are 
likely to increase the reliability and validity of the process. Mul- 
tiple measures will improve the chances that the needs identified 
during the process are indeed legitimate, and should be attended to 
by program planners, {p. 10), 

The descriptions of the vocational/special education studies and. practices 
cited thus far have emphasized the collection of needs information from the 



prospective training audience (e.g», teachers). While the prospective train- 
ing audience is>Jndeed a primary data source for determining program goals, 
multiple audiences are likely to be involved in or affected by a personnel 
preparation program (e»g., program staff, trainers, trainees, agent(s) from 
external funding source(s))» Therefore, these audiences should be identified 

c 

an,d included in the process of goal identification and selection. The reader 
should find the utility standards and guidelines offered \n the Standards for 
Evaluations of Educational Programs, Projects and Materials by the Joint 
Committee on Standards for Educational Evaluation (1981) to be helpful in 
identifying program audiences and for collecting information that is responsive 
to the needs and interests of these audiences. For more comprehensive and 
indepth treatments of this information, the works of Patton (1978, Chapters 5 
S 6; 1980, Part II, Chapters 6 S 7) and Cuba and Lincoln (1981, Chapters 2 
S 10) are recommended. In addition , Xuba and Lincoln (1981) provide a very 
useful review of the problems associated with attempting to define needs in 
contemporary needs^assessi^^ — '"^^^ '"T^ 

In a chapter on strategies for planning and conducting an inservice 
needs assessment, Davis (1980) provided a checklist of activities to be com-- 
pleted. A slightly adapted version of this checklist is shown in Figure 1. Of 
particular note is the emphasis on needs assessment as a continuous process, 
occurring before, during, and following the delivery of the inservice pro- 
gram. ^ 
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Figure 1 

A Checklist of Activities For Steps in tine Needs Assessment Process 



State Concerns 

Identify concerns (problems that suggest a need Tor inservice 
training) . 

Identify target population (individual or group). 



Identify People and Roles 

Determine who will manage the needs assessment. 

Select needs assessment planning team. (Include representative(s) 

frorYi target audience.) 

Identify those who will conduct the assessment. 



Plan the Needs Assessment Data Collection 

Develop a working definition of "needs". 

Determine needs assessment goals. 

Determine needs assessment data collection strategies. 

Determine and obtain' resources required for needs assessment. 

Develop data collection plan and time line. 



Implement the Needs Assessment Data Collection 

Develop instrumentation and recording procedures. 
^ Field test and validate instruments and procedures. 

Collect needs assessment data. 
Tabulate data collected and summarize results. 

^ Report to planning team. Plapning team interprets and judges 

results. 



Disseminate Results and Set Priorities 

Disseminate results of^ needs assessment to respondents and inter- 
ested constituencies. ^ 

Prioritize needs for training. . . • ^ 
Determine feasibility of meeting the needs and select prioritized 

needs for training. 
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6. Design the Inservice Program 



Identify the target audience. 
Identify needs to be satisfied. 

Select training or activity to be offered to satisfy the need. 
Identify who will be responsible for each activity. 
Identify resources needed to accomplish the task, including incen- 
tives to be offered to training participants. 
Identify how progress and accomplishments will be assessed. 



Continue to Asse s s Needs 

Determine strategies for continuous assessment during conduct of 

training prograrp. 

Reassess needs when program has been completed. 

~~ Evaluate progress and accomplishments. 



Adapted from Davis (1980), p. 16. 



Selecting the Program Strategy 

The second purpose of the planning evaluation phase is to help in 
selecting the most appropriate strategy for addressing the identified needs 
and goals. Procedures for developing the "best" program strategy could range, 
from a fulhscale feasibility study to a two-hour meeting with a program 
board. Other potential procedures have been suggested by Brinkerhoff 
(1980): 

1. Critical reviews of literature, journals, reports, texts, 
proposals, etc. — to learn about the history of efforts 
mounted to deal with goals similar to those to be pursued; 

2. Two-phase studies which first would survey pertinent 
literature or other sources to assemble a "catalog" of 
previously conducted program designs and approaches to 
dealing with similar goals. These alternative strategies 
would then be evaluated one against the other, or against 
a set of criteria, to determine their comparative strengths 
and weaknesses, or to rank-order them; 

3. Site visits to (or other kinds of contact with) operating 
programs addressing goals similar to the goals to be 
pursued; 
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i|. Surveys of experts—practitioners and theoreticians~to 
glean their ideas, views, and opinions concerning success- 
ful program strategies; 

5, Convening of "advocate teams", each assigned to develop 
a program that could operate within a given set of re- 
sources and could meet a given set of criteria. The 
program designs produced by these teams are then eval- 
uated by a judge, or panel of judges, to select a "win- 
ner"; and 

6. Convening of meetings, conferences, and public "hear- 
ings" to review and critique certain program, alternatives, 
a given' single program, or particular aspects of programs 
under consideration, (pp. 35-36) 



The listing of the above procedures in a separate manner does not necessarily 
imply that a single procedure is recommended. In practice, the authors have 
observed several training programs that used a combination of related litera- 
ture review, site visits to similar programs, surveys, and discussions with 
experts and feedback from program advisory members for planning evaluation 
purposes. 

The planning evaluation activities noted in this section are most visible 
in the beginning stages of a new program, ,when staff efforts are heavily 
focused on program design matters. Yet, these activities should alsd be 
present, but perhaps less apparent, in the ongoing operation of a personnel 
preparation program. 



) 



Phase II: Process Evaluation 

Process evaluation occurs during the implementation stages of a per- 
sonnel preparation program, peeking to answer the basic question, "How are 
we doing in getting there?" (Wentling, 1980)* It involves collecting and 
sharing information for program improvement purposes. The particujar focus 
and purposes of this type of evaluation will depend on the maturity of the 
program and the information needs of the audiences for the evaluation* 
However, the greatest payoff of process evaluation, particularly for new 
programs, is that problem areas are identified along the way and corrective 
measures can be taken almost immediately. 

To illustrate the varying upes of and approaches to process" eyaluatrdn in 
the teacher education context, three case examples are offered here* These 
examples describe evaluation activities conducted for a short-term, workshop, 
a course, and an inservice project. 

Short-Term Workshop 

A framework for evaluating the effectiveness of short-term workshops 
has been conceived and used by Knowlton (1980)* His framework includes 
"targets across three evaluative conditions: (a) pre-post session, (b) within 
session, and (c) follow-up" ( p* 60)* The "within session" condition focuses 
on evaluative activities conducted during workshop delivery* Knowlton sug- 
gests the evaluative data obtained during a short-term workshop be generated 
from: 

— Probes of participant proficiency relative to knowledge 
and performance; 

Participant reactions to critical logistic factors (e.g*, 
instructional materials, media); and 



— Direct observation of participant attention to workshop 
delivery, 61 ) 

He also advocates the use of a variety of measures for collecting effec- 
tiveness data, such as " paper and pencil surveys, direct observatidn pro- 
cedures^, and participant interviews. 



Course Development ^ 

An evaluation of a newly developed special/ vocational education course, 
offered through the University of Vermont's "Careers" inservice project 
(Hasazi, 1980), was conducted by an external consultant. This consultant, 
or external evaluator, summarized the intent of the evaluation procedure: 
{Center for Evaluation and Policy Research, n.d.) 

The entire 'procedure was an open dialogue between the 

^BWlo^T^wdnhennstractors-;— "The-evaluator^ought-informa-- 

tion applicable to the needs of the instructors for developing, 
teaching, and modifying the course- For this reason a com- 
bination of participant observation, evaluation questionnaires, 
and telephone interviews were used to gather information. 



Descriptions of the three data gathering techniques were also prepared by the 
external evaluator: (Center for Evaluation and Policy Research, n.d.) 

1. Participant Observation , As a participant observer, the 
evaluator attended the first or second class, the eighth 
class, and the last class (#15) of each semester. The 
participant observer took note§ on the content of the 
course as well as the students* actions. Their attentive- 
ness (body language), actions, and comments were noted 
along with' the environmental conditions of the classroom. 
During the break the observer talked with class members 
and obtained information on their satisfaction with the 
course. These notes were not shared with the in- 
structors directly, though they were discussed in general 
terms with them after each visited class. 

2 Midterm^and^^Rinal-Cour-se^Evaluation-^Questionnaire —Evalua- 

tlori forms were developed with input from, the course 
Instructors. Questions the instructors were curious or 
- concerned about were Included in the evaluation. For 
example, the Instructors were very Interested in knowing 
what the participants thought about using three instruc- 
tors to teach the course. As a result, the question was 



Included on the evaluation form. The results of the 
evaluations were reported in median • scores with par- 
ticipant comments listed. The reports were then shared 
with the instructors,.. . 

3. Fo llow-up Phone Calls . Based on the information obtained 
Trom the evaluations, the instructors felt that two or 
three questions merited further research. Three or four 
course participants' names were randomly selected and 
called at their place of employment. He/she^was asked to 
elaborate on the questions the instructors requested more 
information about. These answers were kept confidential 
and written up in a follow-up phone call report which was 
then shared with the instructors. 



Inservice Project 

In an inservice project at the University of Kansas, Skrtic et al. (1981) 
were involved in training teams of vocational and special educators to become 
inservice providers in their respective school districts. The implementation 
phase of this project consisted of two parts. First, the t^am members re- 
ceived training^ via a two course sequence which took place during Spring and 
Summer 1980. Second, the trained team members then deliver.ed "needs- 
based, district-specific inservice programs" (p. 9) to their peers during the 
1980-1981 school year. Da.ta were collected from five sources to determine the 
extent to which the teachers in trainiag were acquiring the skills necessary 
to be inservice providers. These data sources were: 

1 . Staff observations during training sessions; 

2. Staff review of inservice products developed by the trainees; 

3. Formative evaluation contlri'ents made by the trainees; 

Comparisons of final examinations with pretests; ano* 

5^ „^Projej:l.„sJtaf£_-and„,audience--evaluations of- delivered inservice -pro- 
grams by trainees, (p. 34) 

Interestingly, in their pretesting of 'the teacher-in-training group, Skrtic and 
his colleagues found that "few had even an awareness of needs assessment 



and' evaluation principles" (p. 3^)- In reviewing their description of the 
training components and the follow-up technical assistance provided by the 
project staff, it was evident that the evaluation area was an important com- 
ponent of the training program, 

A quote from Stufflebeam et al. (1971) summarizes the intent of process 
evaluation and its potential benefits: 

... under process evaluation, .information is delineated, ob- 
tained, and reported as often as project personnel require 
such information, daily if necessary— especially during the 
early stages of a project. This provides project decision mak- 
ers not only with information needed for anticipating and over- 
coming procedural difficulties but also with a record of process 
information for interpreting project attainments, (p, 232) 



Phase III: Impact Evaluation 

Impact evaluation occurs following the trainee's exit from the program 
and entry or re-entry into employment. This third phase of program evalua- 
tion examines the outcomes and effects of a program, focusing on two central^ 
and closely related questions: (Brethower S Rummler, 1977j 

1» Are the concepts and skills taught in the program being used by 

the graduate in the workplace? 
2. Does application of these concepts and skills positively affect the 
work environment? 

The emphasis here is on changes in the graduate^s performance as well as 

impacts on the workplace. Has, for example, the graduate who received 

training on delivering inservice education initiated an inservice effort at the 

local level (Phelps S McCormick, Note 2)? What effect has the application of 

these procedures had on increasing the participation of handicapped students 

in vocational education? 

While the results of impact evaluation should be useful to the internal 

staff for program planning and improvement functions, impact information also 

tends to capture the interest of people in positions who make judgments about 

* 

whether to continue, expand, or certify a program (Anderson S Ball, 1978) • 
In personnel preparation circles, these decision makers may Be from univer- 
s ity ad ministration,^, siate„. and. .federal funding agencies, or accreditation 
.associations. 

Two case examples of impact evaluation designs in vocational/special 
education personnel preparation projects are highlighted below. 



Maryland Project 

An interdisciplinary career/ vocational education project conducted at the 
University of Maryland (Malouf et aL, 1981) liad two data collection points for 
evaluating project impact:, (a) a three-month follow-up mail survey on "train- 
ing outcomes, new professional activities, and impact on service delivery" (p. 
12); and (b) a one-year follow-up telephone interview with former program 
participants. The questions asked of the graduates in the three-month mail 
survey are displayed in Figure 2, These questions appear to be of a general 
exploratory nature, which seems appropriate given the short time period since 
program completion. 



Figure 2 
Graduate Follow-Up Survey * 

Career/ Vocational Education for the Handicapped (CVEKj 

'A 

1. Due to your participation in the CVEH program, have you changed some 
components of your classroom operation? Yes No 

Explanation : 

2. Have you become involved in any other activities related to CVEH? 

Yes 'No 

If yes, please explain. ' . \ 

3. Are you in a new position? Yes No 

ci) ' ' " ^ 

If yes, is Jthis position related to CVEH? 

If yes, please explain. 

i i 

i». Do you have any additional plans for this year related to CVEH? 
Yes . No 

If yes, please explain. 
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No 



No 



5. Woutd you like a follow-up visit by a mefmber of the CVEH staff? 

Yes No 

If yes, regarding: 

6. Are you interested in further coursework in CVEH? Yes 
If yes, on what topic(s)? 

?• Are you interested in a position focusing on CVEH? Yes 
If yes, what sort of position? 

8, Are you planning to move? Yes . No 

If yes, we would appreciate your new address so we can keep in contact 
with you. 

* 

9. Would you be willing to host a practicum student this Spring? 

Yes No 

♦From University of Maryland Career/ Vocational Education Inservice Project, 
no date. 



The one-year follow-up telephone interview guide developed by the 
Mary,and- project staff is shown in Figure 3. In contrast to the mail survey 
questions, the items in the telephone interview guide appear to be much 
sharper and more focused oh the graduate's impact on the work environment. 
In conducting the telephone interviews, the Maryland project staff used 
separate guides for sp^ecial educators and Ijndustrial arts/vocational education 
personnel . Malouf. the project director, provided a rationale for this decision 
(Note 1): 

The main thing I've learned abtout evaluation in this project is 
that the different training populations-special educators, vocational 
educators, industrial arts educators^-are likely to react, differently 



to many training experiences, and are likely to differ in terms ot 

\ 
\ 
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the impact of training. Therefore, we try to provide for separate 
analysis of the evaluation data according to professional field, and 
to devise separate instruments as needed. 



Figure 3 , ^ 

On e Year Fo l lo w-Up Telephone Interview Guide* 

Special Education Personnel 

We would like to ask you some questions regarding our training sequence 
in Career/Vocationai Education for th^ Handicapped. Please compare your 
present activities with your activities'^ prior to enrolling jn the training se- 
quence. 

1. Have you^ developed a more effective working relationship with the voca- 
tional education 'and/or the industrial arts personnel serving your stu- 
dents? V/ 

a. Have you been able to involve them more effectively in the lEP 
process? 

* 

b. Have you been including more career/vocational goals in lEPs? 



c. 



Are you communicating more information to industrial arts/vocational 
education teachers regarding the handicapped students they serve? 

• <^ 

Are those teachers providing you with suggestions for methods or 
activities you can use in your own teaching? 



2. H^ve you changed any aspects of your direct services to , handicapped 
students? 

a. Are you collecting assessment information on career/ ' vocational* 
interests, aptitudes or independent living skills that you were not 
collecting before? 

b. Are you more effectively providing instruction Jn careers and 
employabjlity skills? 

<■ c. Are you more effectively preparing your stud^ts to function In I A 
and VE- programs? 
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/ d. Do you feel that your students are being better prepared for 
* / employment? 

3, Are you involved in the planning^and development of new programs or 
services for the handicapped students? , 

4. Did" the project contribute to any changes you made in the areas we just 

discussed? (If no, what contributed- to the change?), 

* * . » 

a. Looking, back, what experiences were most beneficial? 

f 

b. What should be changed or omitted? 

c. What e'xperiences should be added or expanded? 

Other Comments: . • * • 



*Frpm University of Maryland Career/ Vocational Education Ins^r^^ice Project, 
''no date, . ^ 



Kansas Project 

The second case example is the University of Kansas project (Skrtic et 
al., 1981) which was cited earlier in the process evaluation section. In the 
second stage of the project, following intensive training, the staff systemat- 
ically observed the trainees ,as they functioned in new roles as inservice 
providers' in their respective school districts. Evaluative data on the in- 
service delivered by the trairj,ees were collected from the inservice partici- 
pants and the project staff, • , 
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'since the inservlce provider ^tivities of the trainees were part of the 
Kansas project, this case example is noj an ac:tuar impact evaluation. How- 
ever, the naturalistic ^observation and data collection approach employed in 
this project is worthy of consideration for impact evaluation purposes. By 
being in the graduate's , work setting, observing her/him "in action", and 
interviewing significant others (e.g., mainstream teachers, students), the 
evaluato'l- could obtain an indepth and rich contextual understanding of pro- 
gram impact. He/she would be gathering first-hand information on the (effects 
of the graduate on the workplace (and vice versa), while also getting a direct 
treading of the extent to which the program goals and expectations are 
"reali.ty based". 

Need for Impact Evaluations 

Reports on the evaluation of program impact are, indeed, rare entities in 
the vocational/special education literature. Of those reviewed, most consisted 
of short term follow-up designs involving, for example, the collection of data 
from graduates within a year following program completion. Impact evalua- 
tions of a more longitudinal nature did not appear in our literature review. 
This is understandable though given the recency of the vocational/special 
,educatioh emphasis in teacher education circles and the complexities associated 
with conducting longer-term impact evaluations (e.g.*, substantial' time and 
cost investment, competing program priorities, difficulty in establishing 
cause-reffect relations over time). Nevertheless, both short and longer term 
studies of program impact have the potential for increasing our understanding 
of the relationship between specialized training and performance in the work 
setting. , The .inclusion, and reporting of impact studies in vocational/special 
education* personnel prepar-ation programs are encouraged. 




A Primary Consideration 



The three-phase perspective of program evaluation presented in this 
paper provides a convenient framework for examining the various purposes 
and procedures of evaluation in the development and operation of a personnel 
preparation program. This framework should be helpful to the evaluator(s) 
and decfsion maker(s) of a vocational/special education pr'ogran when con- 
sidering the focus of an evaluation; that is, when taking into account the 
what, why, when, and how to questions of an evaluation. However, primary 
consideration needs to be given to ways of increasing the likelihood that the 
evaluation .findings will be utilized by the key decision makers and information 
users of the evaluation. The importance of this impact consideration was 
emphasized ^by the Joint Committee on Standards for Educational Evaluation 

(1981): ' * 

It is' crucial to identify the audiences (involved in or 
affected by the evaluation), to rank order them, and within 
resource and time limitations to strive to ascertain and accom- 
modate their information needs. If this is not done, the 
evaluation may be a misguided, academic exeVcise whose results 
are ignored, criticized, or resisted because they do not ad- 
dress anyone's particular questions. On the other hand, an 
evaluation planned and conducted to meet the informational 
needs of identified audiences is more likely to receive a posi- 
tive response, (p. 21) 

The Joint Committee suggested that the criteria for ranking audiences in- 
clude: (a) expressed interest of each potential audience, and (b) their pros- 
pects for using the evaluation to influence or , make decisions affecting the 
program. 

The identification .of key audiences and their information needs should go 
beyond the evaluation design stage. Priorities, , concerns, and information 
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needs of these audiences are likely to shift as the evaluation unfolds. The 
evaluator should, therefore, be sensitive to this possibility throughout the 
evaluation. The Joint Committee oh Evaluation Standards (1981) took this 
notion one stej^ further by offering guidelines for the evaluator to actively 
"ensure that the members of the audience will assess and make constructive 
use of the results of an evaluation" (p, 47), These guidelines are: 

1, Demonstrate to key audiences at the beginning of an 
evaluation how the findings might be useful for their 
work, 

2, Arrange for the involvement of representatives of the 
audiences in determining the questions and planning and 
implementation procedures of the evaluation. 

3, Be open, frank, and concrete in reporting to audiences 
and be available and willing to assist in clarifying the 
reports, 

4, Periodically report interim results, noting especially how 
these may apply to roles performed by members of the 
audiences, 

5, Assess the merits of plausible alternative courses of 
action and discuss those in the final report, 

6, , Suf^plement written reports with ongoing communication. 

7, Within limits of time and resources, plan to help the 
audiences assess, interpret, and apply the findings 
beyond the time when the final report is submitted, (p, 
47-48) 

The procedures and guidelines suggested in the preceding paragraphs 
are ultimately intended to increase the utilization of evaluation findings. In 
discussing the research on utilization, Patton (1978) suggests that we not 
look for major changes to suddenly occur as a result of the evaluation. 
Instead, recognize that changes come. sFowly with "impacts in ripples, not' 
waves" (p, 33), He summarizes this point with a sense of optimism about the 
potential of utilization-focused evaluation, while also recognizing its limits: 
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utilization . of evaluation research can be increased and more 
tarefully targeted, but evaluation findings will seldom have the 
enormous kind of influence envisioned by social scientists who 
vyanted to rationalize decision making processes. The potential for 
enhancing utilization lies less in its capability for rationalizing 
^4eclslon making than in its capacity to empower the users of 
evaluation information, '(p. 35) 
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V 



Parting Observation 

From the experience of preparing this monograph, some parting thoughts 
are offered on one aspect of program evatuation that stands to benefit from 
closer examination in future professional forums, 



The selection of exemplary program evaluation practices for inclusion in 
this publication was somewhat difficult. If exemplary is viewed as being an 
example, selected on the basis of representativeness and for its potential use 
by others in similar areas, then a number of exemplary practices are pro- 
vided. However, a major problem encountered in selecting procedural 
examples and sample instruments was that the efficacy o" the methods used in 
program, evaluation was seldom„examined or at least seldom reported by the 
authors. Selected authors did provide some reflection on their evaluation 
methods, but these were rare exceptions. Judging from the way in which 
many projects/ programs were reported, it seems as though the methods 
selected and used were a "given" in the program evaluation effort. We sub- 
mit, that the importance and influence of the evaluation method employed in a 
project or program warrants closer professional scrutiny. Therefore, it is 
recommended that personnel involved in the evaluation of vocational /special 
education personnel preparation programs be more explicit in communicating 
the rationale behind the selection of particular methods used, the processes 
employed in developing the procedures and related instrumentation, and the 
relative, efficiency and effectiveness of these methods in the context of the 
evaluation activity. This information could be" very useful to colleagues who 
are in the process of mounting a program evaluation effort. Future authors 
of monographs on exemplary program evaluation practices in special/vocational 
education would also be indebted! 
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